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IntrodtictlQn 

The current wide-spread use ojf the semaiitlc differential (SD) for Index'- 
img outcomes of Instruction Is not surprising. The SD Is easy to administer , 
has a large body of literature to support its use, and is relatively easy to 
analyse* Still g few resaarclters or evaluators are aware of the breadth of use 
of the SB. Heeht (1971) trevlewed twenty^some studies using the SD in the 
evaluation of Instructional outcomes in science educatioii alone* Bi--pola'f ad^ 
Jectlve scales have been used to evaluate conferences (Frlsen & Bumbargiir, 1972) ^ 
collage freshman classes (Hoover, Baumann, k Shafer, 1970), Inscructlomal pro^ 
grams (Carver & Phlpps, 1969) p and even Individual lessons (EvanSp 1970). 

But naay studies using SD Instruments to evaluate outcomes of Instruction 
appear to have basic methodological or theoretical difficulties* In general the 
three moat important difficulties are theses 

A*, The studies are undertaken without CGnslderatlon of the ass.umptions 

or imitations of traditional SD theory. 
B- The studies fall to precisely consider what it is that the SD Indexes* 
C, Studies fall to employ appropriate analytical methods. 
The purpose of this paper Is to set out an alternative approach to evaluation 
using the SD, based on a restricted model for SD application which meets the 
practical and methodological requirements of educational evaluators. While this 
restricted model cannot be applied in all situations. It does appear to have 
important advantages in the measurement of affective outcomes of instruction 
in those situations where one is particularly interested in indesclng a general 
evaluative response to the instruction, where the learner population is of moder- 
ate sl^e, and where one is evaluating responses to easily defined concepts. 



The Restrfcted Model 



The major requirement of any SD application Is tliat the researcher respect 
the structural or geometric properties of the SD, In traditional SD analysis 
this means dealing with the conventional iemantic space solution of the analy-* 
sis of a set of bi-polar adjective scales, Osgood^ Sucip § Tannenhaua (1957) 
sat forward a model space made up of three dimensions i &^aluative7 ^otencyp 
and Activity (EPA)* They arrived at this structure by the factor analysis of 
many scales, and argue that it is stable across concepts , people, and even aeroaa 
cultures* The restricted model of affective evaluation pursued here focuses on 
the single dimension of the evaluative factor* While there are several reasons 
for making this restriction, the central argument Is simply that the affective 
response Inforaatlon cQntalried in SD responses appears in that dimension. 

In one of his early studleSj Osgood (1952) found that many of the bl-polar 
scales appjliad to a concept loaded! heavily on a single factor. That factor^ ha 
ffoutidj aecounted for a large proportion of the total variance In the scales^ 
often as much as 70% or more. In addition ^ several Individual scales loaded very 
highly on the first factor. The restricted modal here represents a throw**back to 
that earlier model of the evaluative factor for the purpose of using SD response 
data In the evaluation of affective outcomes* 

The advantage of the single factor model ©f the SD In evaluation being set 
out here is that it helps the investigator solve three basic problems encountered 
in SD evaluation studies, ^ese arei 

1. nie problem of what it is that the SD IndeKes. 

2. The problem of concept-scale interactions. 
V ' 3. The problem of appropriate methodslogy. 

Each of these will be considered in turn. 



What does the SD Index ? An investigator following traditional SD theory 
who has read The Heasurement of Meaning several times will Invariahly respond 
to the question of what the SD indexes with the knee^-jerk reflex: The SB 
ffieasures connotative meanlnR ^ But does ±t? Or^ even more important, is cog-* 
notative meaning what the evaluator wishes to measure when looking at affeetive 
responses to instruction? Probahly not. The question has been debated almost 
constantly for twenty years by critics from Carroll (1959) to Mlron (1969) who 
asks I "What is it that la being differentiated by the semantle differencial?"^ 
Perhaps a better question for evaly^ators is : Islhat is it that we want the 
semantic dlfferentirl to dlfferentiata? One answer which is of importance fDr 
many evaluators is: Affective responses to concepts. 

If one eKamlnes the sorts of scales which are comionly found to load 
heavily on che first unrotaced factor ^ there are many whiah are easily identic 
fled with affective evaluation, toong these are fair^unfalifj kind — cruel^ 
valuable— worthless s and honest— dishonest * Considered alone^ the first factor 
(or E.--factor) may be characterised as representing general affect~a general 
disposition toward the concept being itidexed by che SD* It is sort of an 
evaluative G--f actor corresponding in evaluation to the general Intelligence 
factor found In intelligence testing. Considered by itself, It is of great 
practical Interest to the educational investigator* It provides a single Indeic 
of the general affective response of a set of people toward a set of Eoncepts. 
In certain applications ^ at least, the SD's first factor indexes affect. 

CQncept--Scale Interactions > A difficulty for many users of SD methodology 
is the problem of concept"-scale interaWions* Essentially the problem is that 
in many studies researchers have failed to find the same pattern of scale 
loadings on factors which Osgood found* The ifflportance of this finding is that 
it suggests thati contradictory to Oegood'e position^ that EPA structure is not 



stable. In fact, it is quite cocmon to fiad scales traditionally thought of 
as definitive for the Potency dimensions (active^ — ^passlve) loading at high 
levels on the first factor or E'-factor, Levy (1969) and Helse (1969) straciarize 
the concept-scale interaction literature well, Osgood and Suci (1955) suggest 
that J in fact, there is not a necessary correspondence between the factors 
obtained in an analysis of SD scales and the underlying dimensions of semantic 
space. 

But the concept-scale Interaction problem has practical consequences for 
the Investigator t^shlng to use bl-polar adjective scales to index evaluative 
responses to concepts, Helse (1969) indicates that one cannot simply assume 
that the scale-concept relationships found In the analysis of one set of concept 
will hold for another set o£ concepts as well, ^erefore, the patterti of 
loadings must be re-determined through factor analysis or multl-dlmectslonal 
scaling for each new study. This raises the cempleKlty of SD studies greatly— 
frequently putting them out of the reach of many investigators* 

The use of the restricted model of the SD advanced here helps overcome 
concept--scale Interaction problems In two principal ways. Firsts since one Is 
concerned with only a sliigle factor or single underlying dimension ^ the ques- 
tion of interaction Is limited to a question of which scales load on the first 
factor rather than which scales load on which of several factors. Thus the 
problem Is simplified* Second^ the analysis whl^h needs to be performed to 
answer this question is also simplified* One does not have to use a full factor 
analysis (although factor analytic techniques \^11 be used) to find out which 
acales contribute in ^at way to the underlying evaluative dimension. 

While one cannot argue that the evaluative factor is stable with respect 
to the pattern of scale loadings^ one can at least argue that the problem le : 
much less critical In the restricted model than It Is In the traditional model. 



Appropriate methodolggy ^ If the methods the Invaetlgator employs in ex-* 
amlnlng relationships in the SD data are to be appropriate, they must be 
eoneietent with the notions set out in the previous dlicusslon* Specif icaHy* 
they must i ^ 

1, Consldeif adequately the problem of struer.ure, 

2. Yield an appropriate index of the evaluative dimension of the structure* 
Ideally the methodalogy should also be kept relatively simple. 

The problem of structure is not complex, but a remarkable nmber of studies 
fall to deal ^th the problem adequately. The key issue Is that since one is 
arguing that the SD is a multiple-scale Index of some underlying difflenslon or 
faetor, one must test relationships on the basis of that underlying dimension, 
and not on the scales themselves^ This point deserves ^panslon, 

Considar the question of testing the hypothesis that an instructional treat-- 
ment brought about a change In affective response, to the concept BANANA in a 
social studies unit on Central America, One hopes that the SD can be used to 
determine If a change took place. But if one simply performs a set of ^t^-testa 
on the individual scales used in the analysis^ there are major methodological 
problems. First, one does not taiow which scales Index the general evaluative 
response. One guesses that good-^bad does/ but what about the others? Then 
suppoae that six out of ten tests were significant. Does that indicate six 
Independent slgnif leant results, or Is the investigator simply measuring the 
same underlying relationship six times? A further problem is that since the 
testa are not independent (scales are clearly correlated) tlie likelihood of a 
type I statlsticEl error becomes very larger €ven with a small number of scales* 
In short, the methodological approach. of perforalng repeated tee^s of slgnl^^ 
cance on individual SD scales is bankrupt and should not be employed under any 
^ circumstances,-^ ,■ ■ - .^ ^ 
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Glven^ then, that the investigator must find some way of caleulattng a 
combined index of the underlying relationships in the data, how should he or 
she approach the problem? The concept"Scale interaction problem strongly 
Suggests that one cannot assume relationships between scales and concepts in a 
study based on the analysis of concepts in another study. So some analytic 
technique, such as !ffiS, factor analysis, or cluster analysis must be employed . 
on the data obtained in the evaluation study itself. The two approaches which 
will be outlined here are both based on factor analyrfli,^ although neither is, 
strictly speaking J a factor analytic method. 

The CFRH method . Allen (1974) suggests that the canonical factor, regres- 
sion method (CFRM) Is a sound general method of computing composite Indexes of 
imobserved variables * Alle^ provides an extensive development of the argument 
for using this particular approach, but the features of particular interest in 
this application are that the technique provides maximally reliable and valid 
estitoates of the tmobserved variable, and that the obtained factor is uncor^ 
related with the rasldual variance in the data* In addition the approach has 
practical marlt since the routines used to calculate the IndeK are available 
through standard statistical programs such as SPSS* 

The procedure followed in calculating the scores for the unobserved evalua-- 
tive IndeK using the CFRM are outlined Ir* Figure 1* Essentially one performs 
a factor analysis of the pooled SD data across the scales using Rao* s Canonical 
Method of factor analysis (Rao, 1955) V Using the factor score coe 
obtained by the regression method (Harris, 1967) , one calculates factor scores 
for the pre- and post--inStruction responses obtained for each concept in the 
analysis. Generally several concepts are used in a given analysis. An appro^ 
prlate test of significant differences between means (usually a t-test) Is then , 
used to determine if the values for the general evaluative response index differs 
across the instructional treatment. 
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Insert Figure 1 about here 

^e advMtaga of the CiTCt method Is that the Index employed has several 
d^elrable properties. It is uncorrelated with the residual variance In the^ ^ 
analysis* It can be used to determine what prbportion of the total varimee 
the index represents 5 and it ijicludes the contributions to the score of each 
scale in the analysis* ^ Its mijor dlsadvmitage is that it is computationaliy 
complicated and is perhaps more detailed than most evaluators need. 

Unit weight method . For those situations in whidi the precision and 
statistical niceties of the CFRM method are not required^ a simpler method is 
available. After obtal^tog the loadings of the scales on tiie first unrotated 
factor (and Roa's Canonical Method is again recofflmendad) , one identifies those 
scales which load highest on the factor. In a set of i6-scalea-lt Is cbmmoa 
to find over half with absolute loadings of ,500 or higher. Summing over those 
scales with loadings above some pre^set level (such as .600) will produce a 
composite Index* Even though this IndeK does not have ideal statistical pro^ 
pertlesV ©ne has a strong argument for suggesting that the Index does measure 
general evaluative response. In fact^ the CFKM Index and the unit weight tadex 
should intercorr elate highly. 

An Itoplrical Demonstration , 

The approach set forward above was employed in the evaluation of affective 
outcomes of a college course in Instructional System. As part of a fomal 
program of course design and development evaluation questions regarding the 
affective outcomes of a unit on the management of instruction were raised. 
Specifically the course evaluators asked if the unit brought about changes in 
Che affective response by leamers to key concepts of the instruction, : 



Concepts. Four concepts were Identified for analysis* These were LEARNING 
RESOURCES, A^V MATERIALS, mHAGE^T OF mSmUCTION, and j^mCAN EDUCATION, 
tie last concept was chosen as a result of a student coMent in an earlier 
course evaluation. The others were selected on the basis of the course con- 
tent outline* 

In addition to the four expermental concepts employed in the study ^ four 
"placebo" concepts were also tavestigated, ^ The purpose of us tog these concepts 
was to show that the evaluative response to coneepts not; central to the instruc- 
tion would not change. The four concepts were chosen for their high faoiiliarity 
They were: KITCHEN p ^^IY, MINNESOTA, and MYSELF AS A STUDENT. In a normal 
evaluation study using the SD^ such dum^ concepts would not be necessary. 
They were included here for; the purposes of the empirical demonstration. 

SP instrument . Hie SD instrument ^ployed in the analysis consisted of 
sixteen scales of bi-polar adjectives. toch learner responded to each set of 

Insert Figure 2 about here 

scales for each concept Investigated, One set of synon^oua scales were In-- 
eluded to provlda^an internal check for erratic response patterns . Scales were 
typically chosen because of their meMlngfulness to the evaluation question 
(Carter, Ruggels, a Chaffee, 1968), ^e concept for each set of scales was 
printed by hand at the top of the form, ^ - — 

^e same form was used pre- and post-instruction. Staple Instructions 
were given prior to each administration. The inter-'test interval was approKl^ 
mately two weete. Responses were hand pundied onto computer cards, although 
the use of op-^scan or sense-mark cards would facilitate the data entry process 
in typical educational applications. Finally, all data was analyged using the 
SPSS package. A staple, original FORTRAN program for calculating a reliability 
coefficient was used. ^ 
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The Analysis . Ttim canonical factor analysis of the pooled scale data 
yielded a large first factor. Unrotated it loaded hi^ly on siaveral scales 

toserf Figure 3 about here 

including bad-^-^good and useful"U5eless at r = -.835 and r ^ .911 respectively* 
These two eKtremely high loadings clearly mark the dimension which the first ■■ 
factor indeKes as an evaluative and/or utility dimension. Four of the scales^ 
_ reflecting difficulty and- speed , had low load utge and low comsunallty eettaates 
associated with them r In effectj then 5" twelve of the s to teen scales loaded 
heavily on the factor* Somewhat surpr is togly^ scales such as active— passive » 
tAlch are normally thought of as not evaluative^ loaded very heavily on the 
first factor. This finding lends support within the study for the concept/scale 
dtateraction criticism of the SD, 

Factor score coefficients were calculated using the least-squares regres^ 
sion method (Lawley If Ma^ellj 1971), Using these weights , a co^ined indeK 
was calculated for the unobserved evaluative variable. One variable was calcu^ 
laCed for each of the concepts. 

Reliability and validity of the IndeK . Prior to using the composite Index v 
for testing the hypothesis that shifts in evaluative response had occurred, the 
reliability and validity of the composite index was detemlned for th*p. pooled 
response data. The approach taken was to determine the Omega reliability cbef-* 
flcient due to Helse Bohmsted (1970), A discussion of the fomula and the 
simple program used for calculating the coefficient are given In Figure 4, 

Insert Figure 4 about here^ 

Very briefly, the Omega coefficient is an index of the proportion of variance = - 
:;ln the composite measure which is conmon variance. As such it is an estimate of 
; the correlation between the composite ijidex and the true value of the unobserved 
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For the pooled response data, the reliability of the composite Inda^c is 
calculated to be n ^ .9494 This suggests that more than 94% of the variance 
observed la the composite index was true score variance of Che unobserved vari— 
able* The validity coefficient corresponding to the Ctaega reliability coeffi=- 
cient is Hho- It is simply the square'-root of the correlation coefficient and 
represents the estimated correlation between the composite IndeK and the unob^ 
eerved variable. That is calculated to be p ^ *974# 

Tests for treatment effects . To test the hypothesis that the instructional 
treatmMt produced changes in the student's evaluative responses to the concepts, 
a paired t--test was performed on the pre- and post-Instruction 'composite indexes 

Insert Figure 5 about here 

for each concept* Among the eight tests performed, only two were significant* 
The responses to the concepts A-V Material and American Education showed sig-^ 
nif leant changes in mean value across the instructional treatment* Within the 
context of Che evaluation study in which the tests were cdnducted, this finding 
was interpreted to mean that attitude shifts did occur, but that the shifts 
were not general to all concepts of the instruction* 

A broader interpretation given to the obse^ations at the time was that 
students had formed negative values regarding the way in whidti American Educators 
used audio^visual materials in general. 

The simpler approach to analysis of the response data^ Involving pooling 
of response data from scales which appear to load heavily on the first factor, 
was not pursued in this study* In general, if the CFRM method is performed it 
is the approach of first choice and obviates the necessity of using the less ac- 
curate but simpler method. It is not likely that any significant shifts in'^ . 
evaluative response would have been observed using that method. 

12 U 
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^ NQtes fog -further development * The restricted approach to using the SD 
in educational evaluation^ then, is presented as a more sound method than is / 
characteristically employed in representative studies in the literature* In 
addition to the overall simplicity of the approach one achieves the toportant.^ ^ 
advantages of loiowimg what is being measured— evaluative response^and . of relat** 
ing to the essentially geometrical properties of the SD* In the restricted 
model one avoids the pitfalls on either extreme* Perfoming tests on individual 
scales is simply Inappropriate; a full— bloro e&nantlc space model is too complex^ 
and imposes too many restrictions on the analysis to be useful to most evalua tors* 
Certainly there are other models* Mapiire (1973), Evans (1970), and Stigglna 
(1972) suggest what some of them might be • But for the immediate purposes of 
determining changes In values associated with concepts due to Instructional ; 
treatments the restricted model set out here appears to meet the needs of a 
broad rasge of evaluation applications* - 

i _ , - 
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Figure 1 . -'■i-rP-:.'}'.\^[\ 

Visualization of the Canonical Factor-Regression Method Analysis 
Procedura Employed in Testing Hypothesis : One . ^ v ^^^^^^^^^ 



Pre^instruction 
SD response data 



Pos t-^ins true tion 
SD response data 




Pooled; SD response 
data # all concepts 
all treatments ^ 



Factor • Analys is 



Exploratory extraction of 
first factor using Raofs^- 
Canonical method of 

maximum^riklihood factor ; 
analysis (CF) . ^ ^ 



o 



Calculation of factor score 
coeff icients for vthe ^ first " 
factor using regression : 
method (Rfi) 



Calculation of composite -evaluation 
firs 1 1 factor , i ndex ^or ^ each tre a tmeri t ^ 
each concept using regressed waights^- 



Test for significant differenca ^ . 
between means ( t-test) across instruc- 
tional treatment for each concept y_ . 
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Figure 2. 



Samantle Dif f arential Ins truraent Used 
In Evaluation of Affectiva Outcomes 



worthless : 
knowable t 
aas7 s 
bad z 
fast I 
m^ll t 
exciting s 
orderly : 
J active : 
Imterastlng r 
confusing 2 
single :^ 
strong : 
^ : useful r 
effortleee r 
l^ortmt :^ 



valuable 
_2 unlmow^la 
_i difficult 
_t good 

slow 
^: large 
J borteg 

clutered ' 
J passive 
J dull 

J understandable 
J difficult 

weak 
_: useless - 

demanding 
J unliE^artant 
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\ Figure 3 . 



Generation of a eomposita Index of an unobserved / 
evaluative fae tor based on maxiffiiai--likelihQod faetor 
analyaig of sixteen scales of the smantic differential* 



Scale 


Loading 




? 


Weight 


worthless=val- ■ 


-.825 


,656 


.334 


-.122 


knowab le=unlm * 


-637 


. .443 


.557 


, .058 


easy— difficult 


-127; 


,019 


.981 


.007 


bad-^good : 


-".835 


.656 


.344 


-.123 


f ast=slow . V 


-497 


.272 


.728 


.035 


HBiall=large 


--72B- 


.547 


.453 . \ 


-.082 


^citing— boring> 


-784 - 


.613 


.387 


. .103 


orderly=clutt * 


-660 


.469" 


.538 ; 


.053 


aetlve-^pas^lve 


-769 


.592 


.408, 


.096 


lnteresting~d. . 


-769 


.596 


.404 


.096 


conftising~und^ ^ 


--705 


.524 


.476 


: -.075 


simple— difficult 


,085 


.009 


.991 


.004 . 


strong— weak 


-692 


.501 , 


. .499 


■ .070 


us ef ul— u s ele s s - - 


-912 ; 


.735 


, -265 


.175 


effortless— dman; 




.134 


.866 


-.021 • 


l^ortant^unimb V 


.816 / 


.636= 


.365 


.114 














- . — Figure ~4. ' 
Calculation of Factor Weights , Reliability , and. Validity 



Weights : 



60. = 
1 



f ./ u." 

1 1 



1 + E(f^./U^^) 

1 1 



where is the linear weight for scmle 

is the factor loading between scale ^ and the factor , C 
is the unlquenasa associated with scale 1 ( I'-h ) • 



Reliability I 



it - 2(f ,01.) 

X 1 



where H la the Omega reliability coeeficient. 



Validity: 



where P is the Eho validity eoefflciant. 



FORTEM . Program for Computing Above Values 

PROGRAM RELY (INPUT, OUCTUT, TAPES) 
MANSION F(50), HC50), 1J2(50), WE(50) 
READ (8,101) NUMSAL 
101 FORMAT (12) 
SIGMA ^ 0 

DO 10 I = l,NimSAL 

^AD (8,103) F(I),H(I) 
103 FORMAT^ (2F6,0) 

U2(I) = 1 - H(I) 
10 Simk - SIGMA + F(I)**2/U2(I) 

DENOM = 1 + SIGMA 
- DO 20 1 ^ 1, NUMSAL 
20 WE(I) - (F(l) / U2 (I)) /DENOM 
- "^ ^ RELY SIGMA/DENOM ° - ^ 
- VALID ^ SQRT(RELY) 



PRINT 105 
105 FOimAT IHl 

DO 30 I - i,Nmis^' ^ 

PMNT 107, F(I), H(I), U2(l),~ra(I) 
107 FOm-lAT (2X,4F10:3) 
30 raNTINUE 

PRINT 109, RELY, VALID 
109 FOWIAT (//2X,14HRELIABILITY ^ , 
+F6,'3/2X,14HVALIDITY ' _ _ . ^ : t"^ 
+F6.3) 

CALL EXIT , " , \ • 

. - END - ' — ^ — ^ — 

-EOR- - 
*card~wlth number "of scales "in; col 1-2^ 
*data cards, one per scale, with 
scale weight punched in col 1-6 and 
the uniqueness punched in 7-12* 



Figure 5, . ' . , , 

Comparison by t-^test for paired observations of cDmposite factor 
scores for an evaluative index between pr©'-^ and pos t-iniitructioniil 
administrations of 16 semantic dif f erential scalas on eight 
concepts, (N - 37 and df 36 for all cases) , 









m 


■ m 


jje-Lca i b j 








pi£— caii; 


1. 


KITCHEN 


-.307 


.132 


.022 


.208 


.094 


.016 


1.17 


.251 






(-.325) 


(.132) 


(.022) 












2. 


LEAHNING 


-.360 


. Ill 


.018 


.008 


.067 


.011 


.70 


.490 




RESOURCE 


(-.368) 


(.116) 


(.019) 












3. 


hBm 


-.152 


.200 


. .033 


.006 


.083 


.014 


,44 


.660 






(-.158) 


(.215) 


(.035) 












4. 


A-V 


-.359 


.103 


.017 


.020 


.051 


.008 


2.32 


.026 * 




MATERIAL 


(-.379) 


.111 


. .018 












5. 


MINNE-- > 


-.327 


.163 


.027 


.004 


.066 


.011 


.33 


.744 




SOTA 


(-.331) 


(.151) 


(.025) 












6. 


mNAGE-- 


-.310 


.121 


.020 


.00 8 


.064 


.011 


.78 


.440 




mm 


















7. 


MYSELF 


-.282 


.184 


.030 


-.007 


.096 


.016 


-.43 


.671 




AS A , • . 


(-.276) 


(.178) 


(.029) 












8. 


AMERICAN 


-.246 


.19 0 


.031 


1.390 


.861 


.141 


9.82 


.001 *** 




EDUCAT. 


(-1.636) 


(1.02) 


(.168) 













NOTE i Niintoers in parentheses refer to post*-ins data, 
nundbers above refer to pre-instruction data- 
*Result significant with p .05. 
***Results significant with p. .001. 
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